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Gregory  K.  W.  K.  Chung,  Ravi  Sinha,  Adriana  A.  de  Souza  e  Silva, 
Alicia  M.  Cheak,  Joanne  K.  Michiuye,  Farzad  Saadat,  William  L.  Bewley, 

and  Eva  L.  Baker 
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Abstract 

Effective  delivery  of  advanced  distributed  learning  (ADL)  training  to 
individuals  in  a  Naval  environment  requires  tools  to  support  the  creation  and 
administration  of  assessment  tasks.  We  have  developed  a  knowledge  mapping 
authoring  system  intended  to  be  simple  and  user-friendly  in  its  interface,  but  highly 
functional  requiring  a  minimal  number  of  clicks  to  navigate.  The  authoring  system  is 
intended  to  allow  a  diverse  set  of  users  to  create,  modify,  adapt,  and  reuse 
knowledge  mapping  tasks.  Various  scoring  options  provide  information  on  student 
performance  on  different  dimensions  and  at  different  levels  of  stringency.  The 
system  stores  student  data  and  thus  performance  can  be  monitored  over  time.  This 
report  introduces  knowledge  mapping  and  provides  guidelines  on  the  creation  of 
knowledge  mapping  tasks,  and  then  describes  the  operation  of  the  CRESST  Human 
Performance  Knowledge  Mapping  Tool  Authoring  System. 
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Introduction 

The  Navy-wide  distributed  learning  training  vision  is  to  deliver  quality 
training — to  the  right  people,  at  the  right  time,  and  at  the  right  place — as  part  of  a 
career-long  training  continuum  supporting  Navy  operational  readiness  and 
personal  excellence.  Achieving  this  vision  of  delivering  learner-centric  training  to 
individuals  will  require  maximizing  the  overlap  between  instruction — its  format, 
pacing,  and  complexity — and  the  individual's  cognitive,  affective,  and  learning 
predispositions  and  preferences,  deficiencies,  and  strengths.  Attaining  this 
capability  will  help  equip  Sailors  and  Marines  with  effective  mission-essential 
competencies  when  and  where  needed  at  an  affordable  cost. 

A  critical  first  step  in  developing  learner-centric  systems  is  gathering  quality 
information  about  an  individual's  competency  in  a  skill  or  knowledge  domain.  Such 
information  includes,  for  example,  an  estimate  of  what  trainees  know  prior  to 
training,  how  much  they  have  learned  from  training,  how  well  they  may  perform  in 
a  future  target  situation,  or  whether  to  recommend  remediation  content  to  bolster  a 
trainees'  knowledge. 

One  tool  designed  to  assess  a  trainee's  understanding  of  a  content  domain  via 
graphical  representation  is  the  CRESST  Human  Performance  Knowledge  Mapping 
Tool  (HPKMT),  which  requires  trainees  to  express  their  understanding  of  a  content 
area  by  creating  knowledge  maps.  Previous  reports  have  covered  the  development 
and  functionality  of  the  HPKMT  (Chung,  Michiuye  et  al.,  2002,  2003).  This  report 
presents  an  overview  of  how  to  create  knowledge  mapping  tasks  in  general,  as  well 
as  a  review  of  the  CRESST-developed  authoring  system  used  to  create  tasks 
delivered  by  the  HPKMT.  In  addition  to  creating  tasks,  the  authoring  system  is  used 
for  user  administration  (e.g.,  creating  users  of  the  HPKMT  and  assigning  knowledge 
mapping  tasks  to  them)  and  for  automated  scoring  of  maps. 

Introduction  to  Knowledge  Maps 

Research  on  expert  knowledge  structures  suggests  there  exist  qualitative 
differences  between  how  experts  and  novices  categorize  and  represent  a  domain. 
Experts  tend  to  organize  their  knowledge  in  a  principled  manner,  resulting  in 
knowledge  structures  that  are  semantically  and  syntactically  richer,  more  detailed, 
and  more  interconnected.  A  cohesive  and  integrated  knowledge  structure  enables 
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better  access  not  only  to  background  knowledge  but  to  more  structured  knowledge 
for  learning  and  thinking  about  new  concepts.  By  contrast,  novices  tend  to  rely  on 
surface  features  which  lead  to  more  superficial  relationships  between  concepts  and 
to  poorly  connected  knowledge  structures.  Research  also  suggests  that  as  novices 
become  more  expert-like  in  conceptual  understanding,  so  too  do  their  corresponding 
knowledge  structures  (Chi,  Feltovich,  &  Glaser,  1981;  Gobbo  &  Chi,  1986). 

One  way  for  trainees  to  express  their  understanding — and  hence  one  source  of 
data  for  assessing  their  understanding — is  for  them  to  create  knowledge  maps  of  the 
domain  of  interest.  Knowledge  maps  are  network  representations,  where  nodes 
represent  concepts  and  labeled  arcs  describe  how  concepts  are  related.  The  basic 
unit  of  meaning  is  the  concept-link-concept  tuple,  also  called  a  proposition.  There 
exist  various  knowledge  map  formats.  For  example,  a  declarative  knowledge  map  is 
a  representation  of  facts,  principles,  or  concepts  in  a  given  area.  A  procedural 
knowledge  map  represents  a  process  or  procedure.  In  this  case,  the  nodes  would  be 
steps  in  the  process,  and  the  arrows  indicate  the  order  of  these  steps.  Procedural 
maps  may  involve  solving  problems,  forming  plans,  or  making  decisions  and 
arguments.  A  physical  knowledge  map  represents  spatial  or  geographical  placement 
of  concepts  and/or  relationships.  For  example  students  might  be  asked  to  put  in 
place  all  the  states  which  make  up  the  United  States  to  demonstrate  understanding 
of  geography. 

We  have  developed  and  tested  the  use  of  knowledge  maps  (primarily 
declarative)  for  assessment  purposes  in  a  variety  of  domains  and  populations  (e.g., 
rifle  marksmanship  with  Marines;  risk  management  with  U.S.  Navy  engineering 
duty  officers;  physiology  concepts  with  elementary  school  students;  environmental 
science  with  middle  school  students;  genetics  with  college  students;  U.S.  history 
with  high  school  students;  contaminant  transport  systems  with  advanced  civil 
engineering  students). 

The  following  presents  how  a  declarative  knowledge  map  could  be  constructed 
in  the  domain  of  rifle  marksmanship.  In  the  domain  of  rifle  marksmanship,  for 
example,  the  statement.  Aiming  process  is  a  fundamental  part  of  marksmanship,  can  be 
simplified  and  represented  in  the  following  way: 

part  of 

- ► 


Aiming  process 


Fundamentals  of 
marksmanship 
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In  the  above  proposition.  Aiming  process  and  Fundamentals  of  Marksmanship  are 
two  concepts  related  to  each  other  through  the  link  part  of. 

Each  proposition  is  read  as  its  own  complete  thought.  In  order  to  introduce  a 
related  idea  another  proposition  must  be  created.  For  example,  Breath  control  can  be 
associated  with  Aiming  process  through  the  link  helps, 


Aiming  process 


Breath  control 


part  of 


Fundamentals  of 
marksmanship 


and  to  Fundamentals  of  Marksmanship  through  the  link  requires, 


Additionally,  concepts  can  be  related  to  one  another  in  several  ways,  i.e.,  by 
changing  the  direction  of  the  link.  In  general,  it  is  recommended  to  choose  one 
descriptor  which  best  describes  the  relationship  between  two  concepts. 


part  of 

Fundamentals  of 
marksmanship 

Aiming  process 

- ► 

includes 

There  are  two  general  perspectives  on  how  knowledge  maps  can  be  organized. 
The  first  perspective  reflects  a  hierarchical  organization  where  the  hierarchical 
structure  is  inherently  meaningful.  The  second  perspective  reflects  a  network 
organization  where  meaning  is  based  on  the  collection  of  relationships  among 
concepts. 

Hierarchical  organization.  As  concepts  and  links  accumulate  in  a  knowledge 
map,  the  map  begins  to  take  on  a  structure  that  is  both  semantically  richer  (more 
meaningful  links)  and  better  integrated  (more  connections).  As  defined  by  Ausubel 
(1963),  learning  is  the  integration  of  new  ideas,  or  concepts,  into  preexisting 
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knowledge  structures.  Concepts  are  represented  in  a  hierarchical  fashion  with  the 
most  inclusive,  most  general  concepts  at  the  top  of  the  map  and  the  more  specific, 
less  general  concepts  arranged  hierarchically  below.  This  approach  is  especially 
useful  for  well-structured  disciplines  such  as  the  sciences. 

Evaluation  of  knowledge  maps  from  this  perspective  was  specified  by  Novak 
and  Gowin  (1984).  Novak  and  Gowin's  rubric  is  one  of  the  earliest  and  most 
commonly  used  methods  of  scoring  knowledge  maps.  Their  method  considers 
hierarchy  as  an  important  component  of  the  scoring,  as  well  as  propositions,  cross¬ 
links,  and  examples.  In  terms  of  hierarchy,  credit  is  given  for  each  hierarchical  level 
showing  subordinate  concepts  at  a  lower  level  as  more  specific  than  their  parent 
concepts.  Each  valid  and  meaningful  proposition  is  also  credited,  as  are  examples 
and  cross-links.  Cross-links  are  links  between  different  hierarchical  levels.  Novak 
and  Gowin's  scoring  scheme  is  weighted  heavily  toward  the  hierarchical  structure  of 
the  map.  The  theoretical  rationale  for  Novak  and  Gowin's  scoring  scheme  is  based 
on  Ausubel's  theory  of  learning,  particularly  the  idea  of  subsumption  (new  ideas  can 
be  subsumed  under  more  general  concepts)  and  progressive  differentiation  (as 
learning  occurs,  there  is  more  differentiation  among  the  concepts,  which  is  shown 
by  the  inclusion  of  more  propositions  and  cross-links). 

Evidence  from  several  studies  suggests  that  Novak  and  Gowin's  (1984)  scoring 
scheme  can  differentiate  between  high-  and  low-knowledge  students  in  biology 
(Markham,  Mintzes,  &  Jones,  1994)  and  between  first-year  and  advanced  pediatric 
residents  studying  seizures  (West,  Pomeroy,  Park,  Gerstenberger,  &  Sandoval,  2000). 
This  scoring  scheme  also  appears  to  be  sensitive  to  learning,  as  student  map  scores 
increased  over  instruction  (Pearsall,  Skipper,  &  Mintzes,  1997;  West  et  al.,  2000). 

Network  organization.  The  second  perspective  on  how  to  organize  knowledge 
maps  is  based  on  associationist  memory  theory  (Deese,  1962,  1965),  wherein 
meaning  is  represented  as  associations  among  concepts.  Under  this  model, 
structures  are  not  limited  to  a  hierarchical  framework,  but  instead  allow  for  a  variety 
of  relationships  among  concepts. 

Evaluation  of  knowledge  maps  from  this  perspective  is  typically  performed  in 
one  of  two  ways:  (a)  scoring  each  proposition  on  its  quality  using  a  rubric,  or  (b) 
comparing  each  student  map  against  a  criterion  map,  proposition  by  proposition. 

Proposition  quality  scoring  typically  considers  only  the  propositions  contained 
in  the  map  and  not  its  configural  properties.  Each  proposition  is  evaluated  in  terms 
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of  its  quality.  For  example,  Osmundson  and  colleagues  used  a  proposition  quality 
score  as  one  measure  of  the  quality  of  students'  knowledge  maps  (Osmundson, 
Chung,  Herl,  &  Klein,  1999).  Each  proposition  in  a  student's  map  was  scored  on  a  4- 
point  scale,  ranging  from  0  (invalid /illogical)  to  3  (reflects  scientific  understanding). 

Proposition  quality  scoring  methods  have  been  found  to  differentiate  high- 
knowledge  students  from  low-knowledge  students  (e.g.,  Ruiz-Primo,  Schultz,  Li,  & 
Shavelson,  1997a,  1997b).  Scores  have  been  shown  to  correlate  moderately  with 
other  measures  of  content  knowledge  in  other  formats  (e.g.,  essays,  multiple  choice 
tests),  classroom  end-of-unit  tests  and  standardized  tests  of  reading,  math,  and 
science  (Rice,  Ryan,  &  Samson,  1998),  and  physics  problem  solving  (Austin  &  Shore, 
1995). 

The  second  scoring  method  for  network -based  knowledge  maps  is  to  compare 
a  student's  map  against  a  criterion  map.  The  basic  measure  is  the  number  of 
propositions  in  the  student  map  that  are  also  in  the  referent  map.  Example  referents 
include  a  domain  expert's  map,  a  composite  map  of  experts,  or  the  instructor's  map. 

The  utility  of  a  referent-based  scoring  approach  is  that  it  is  efficient  and  it  has 
been  validated  under  a  variety  of  conditions.  For  example,  Ruiz-Primo,  Schultz,  Li, 
and  Shavelson  (2001),  in  addition  to  using  proposition  accuracy  scores,  also  scored 
students'  maps  against  an  expert's  map.  The  correlation  between  the  proposition 
accuracy  score  and  expert-based  score  was  sufficiently  high  for  Ruiz-Primo  et  al.  to 
conclude  that  an  expert-based  method  was  the  most  efficient  scoring  method  (i.e.,  in 
terms  of  scoring  time  and  reliability  of  scores).  Similar  results  were  found  by 
Osmundson  et  al.  (1999)  and  Chung,  Harmon,  and  Baker  (2001). 

In  general,  scoring  student  knowledge  maps  using  expert-based  referents  has 
been  found  to  discriminate  between  experts  and  novices  (Herl,  1995;  Herl,  Baker,  & 
Niemi,  1996),  discriminate  between  different  levels  of  student  performance  (Herl, 
1995;  Herl  et  al.,  1996),  relate  moderately  to  external  measures  (Aguirre-Munoz, 
2000;  Herl,  1995;  Herl  et  al.,  1996;  Klein,  Chung,  Osmundson,  Herl,  &  O'Neil,  2002; 
Lee,  2000;  Osmundson  et  al.,  1999),  detect  changes  in  learning  (Chung  et  al.,  2001; 
Osmundson  et  al.,  1999;  Schacter,  Herl,  Chung,  Dennis,  &  O'Neil,  1999),  and  be 
sensitive  to  language  proficiency  (Aguirre-Munoz,  2000;  Lee,  2000). 
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Guidelines  for  Creating  Knowledge  Maps 

The  creation  of  knowledge  mapping  tasks  has  7  steps  (see  Table  1  for 
summary): 

1.  Select  domain  area  and  specify  learning  objective.  Since  concept  map 
structures  are  dependent  on  the  context  in  which  they  will  be  used,  it  is  best 
to  identify  the  learning  objective,  or  the  particular  problem  or  situation  one 
is  trying  to  understand,  e.g.,  the  fundamental  elements  of  rifle 
marksmanship,  the  mechanics  of  inheritance,  how  photosynthesis  works, 
features  of  different  physiological  systems  and  how  they  interact,  or  the 
factors  behind  the  1930s  Depression. 

2.  Identify  key  ideas.  With  the  learning  objective  in  mind,  experts  review 
curriculum  material  and  generate  lists  of  the  most  important  main  ideas. 
Experts  can  be  the  course  instructor  or  a  designated  content  specialist.  A 
rank  order  of  the  list  is  established  from  the  most  general,  most  inclusive 
concept  to  the  most  specific,  least  general  concept. 

3.  Construct  preliminary  map.  Using  the  list  of  concepts,  experts  construct  a 
preliminary  concept  map,  linking  concepts  with  links.  Note  that  concepts 
are  usually  nouns,  and  links,  verbs.  See  the  next  section  for  guidelines  on 
selecting  links. 

4.  Review  maps.  Check  the  map  to  ensure  all  concepts  are  depicted  and  that 
the  relationships  between  concepts  are  meaningful  and  complete.  Check 
also  for  overall  organization  of  the  maps  for  density  (number  of  links),  level 
of  complexity,  and  interconnectedness  (that  concepts  are  interrelated,  i.e., 
no  concept  is  isolated). 

5.  Modify  maps  according  to  student  level.  Adjust  the  maps  according  to 
student  level.  Will  students  be  able  to  understand  the  meaning  of  the 
concepts  and  links  or  do  the  terms  have  to  be  simplified  for  student 
comprehension? 

6.  Final  list  of  concepts  and  links. 

7.  Final  knowledge  map.  Experts  create  final  map  based  on  revised  list  of 
concepts  and  links.  This  process  should  be  much  quicker  than  creating  the 
preliminary  map. 
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Table  1 

Summary  for  Developing  a  Knowledge  Mapping  Task 
Step  Procedure 

1  Select  domain. 

2  Experts  identify  key  concepts  within  that  domain,  i.e.,  major  ideas 
and  more  specific,  associated  ones. 

3  Experts  create  preliminary  map  with  links. 

4  Expert  maps  compared  and  reviewed. 

5  Concepts  and  links  are  modified  according  to  student  level. 

6  Create  final  list  of  terms  and  links. 

7  Experts  create  final  knowledge  maps. 


Guidelines  for  Selecting  Links 

According  to  Jonassen  (1996)  the  most  difficult  part  of  semantic  networks  is  the 
linking  process.  Good  links,  which  are  usually  verbs,  describe  not  only  precisely  but 
completely  the  nature  of  relationships  between  all  the  ideas.  And  because  ideas  can 
be  related  to  one  another  in  several  ways,  and  on  different  levels,  it  might  often  be 
necessary  to  either  select  the  more  meaningful  link  or  have  more  than  one  link  in 
different  directions  between  concepts. 

The  following  is  a  list  of  guidelines  for  link  selection  (Jonassen,  1996). 

1.  Preciseness  and  succinctness.  Try  to  avoid  surface  links,  such  as  is 
connected  to,  is  related  to,  or  involves,  for  they  do  not  tell  anything  meaningful 
about  the  relationship.  Select  instead  links  that  discriminate  meaningful 
differences  on  functional,  temporal,  or  causal  levels.  For  a  list  of  relational 
categories,  see  Table  2. 

2.  Parsimony.  Try  not  to  use  more  links  than  are  necessary.  For  example,  if  5 
different  links  will  describe  all  the  relationships  among  the  terms,  do  not 
use  more  than  5.  Also,  do  not  use  different  links  that  mean  the  same  thing, 
e.g.,  attribute  of,  property  of,  and  characteristic  of. 

3.  Consistency.  The  meaning  of  any  link  should  be  the  same  each  time  it  is 
used. 
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4.  Avoid  over-reliance  on  one  or  two  links.  A  predominance  of  a  few  links 
reflects  a  narrowness  of  thinking.  Additionally,  it  implies  the  links  are  too 
general  and  that  other,  more  specific  links  might  better  describe  the 
relationship  between  concepts.  One  strategy  is  to  calculate  the  proportions 
of  relational  categories  among  the  links,  i.e.,  frequency  of  causal, 
characteristic,  functional,  etc.,  to  ensure  a  balanced  representation. 

5.  Calculate  temv.link  ratio.  There  should  be  fewer  links  than  terms.  This  goes 
back  to  the  idea  of  parsimony. 

Although  the  HPKMT  provides  no  direct  measures  of  the  quality  of 
propositions  in  the  knowledge  maps,  prior  CRESST  research  has  examined  the 
semantic  quality  of  links  used.  This  approach,  based  on  existing  work  on  semantic 
and  language  structure  (Evens,  Litowitz,  Markowitz,  Smith,  &  Werner,  1980;  Sowa, 
1984;  Wilkins,  1976),  has  identified  nine  common  relational  categories  for  knowledge 
map  links  (see  Table  2,  Chung,  Baker,  &  Cheak,  2002). 


Table  2 

Relationship  Categories  For  Knowledge  Maps 


Relationship 

category 


Definition3 


Example 


Causal  X  creates  a  change  or  effect  on  Y 

Characteristic  X  is  an  inherent  feature  or  characteristic  of  Y 

Classification  X  is  a  class,  category  or  type  of  Y,  or  vice  versa 

Comparison  X  involves  a  comparison  in  order  to  show  a 
similarity,  difference,  or  equality  with  Y 

Conditional  X  contingent  on  Y;  a  possible  event 

Function  X  designed  for  or  capable  of  a  particular  function 
with  regard  to  Y 

Location  X's  spatial  relation  to  Y 
Part-whole  X  is  contained  within,  or  a  part  of  Y 
Temporal  X's  time  relation  to  Y 


causes,  leads  to,  increases,  improves 
has,  is 

type  of,  example  of 

similar  to,  different  from,  equal 

may  lead  to,  requires,  necessary  for 
controls,  transports,  carries,  use 

under,  over 

part  of,  belongs  to,  made  of,  includes 
beside,  during,  follows,  prior  to 


aGeneral  form:  X  type-of-relationship  Y,  where  type-of-relationship  is  the  relationship  category. 


In  considering  the  relationship  between  concepts,  first  determine  the  nature  of 
the  relationship  [e.g.,  what  kind  of  thing  is  it?  (membership);  What  is  it  made  of? 
(whole/part);  What  are  its  distinguishing  features?  (characteristic);  What  does  it  do? 
(functional)]  and  then  select  the  appropriate  descriptor,  e.g.,  is,  made  of,  has,  controls, 
respectively.  For  an  extended  list  of  possible  links,  see  Table  3. 
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Table  3 

Sample  Links  (Adapted  from  Jonassen,  1996) 


Relationship  category 

Examples 

Symmetric 

has  sibling 

is  same  as 

has  synonym 

is  independent  of 

is  opposite  of 

is  equal  to 

Asymmetric 

is  near  to 

is  similar  to 

is  opposed  to 

1.  Inclusion 

composed  of/ is  part  in 

contains /is  contained  in 

(typically  the  most  common) 

has  part/is  part  of 

has  instance/is  an  instance  of 

has  example /is  example  of 

includes/is  included  in 

2.  Characteristic 

has  characteristic/is 

has  attribute/is  attribute  of 

(second  most  common) 

characteristic  of 

has  type  /is  type  of 

has  property/is  property  of 

defines /is  defined  by 

has  kind /is  kind  of 

models/is  modeled  by 

describes /is  described  by 

implies /is  implied  by 

denotes /is  denoted  by 

has  disadvantage/is  disadvantaged 

has  advantage/is  advantage  of 

has  size/is  size  of 

has  function /is  function  of 
is  above /is  below 

is  higher  than/is  lower  than 

3.  Action 

causes/is  caused  by 

used/is  used  by 

solves /is  solution  for 

exploits/is  exploited  by 

decreases/is  decreased  by 

increases/is  increased  by 

destroys /is  destroyed  by 

impedes /is  impeded  by 

influences /is  influenced  by 

determines /is  determined 

enables/is  enabled  by 

absorbs /is  absorbed  by 

acts  on /is  acted  on  by 

consumes/is  consumed  by 

converted  from/converted  to 

designs/is  designed  by 

employs /is  employed  by 

evolves  into/is  evolved  from 

generates/is  generated  by 

modifies/is  modified  by 

originates  from /is  origin  of 

provides /is  provided  by 

requires/is  required  by 
sends  to /receives  from 

regulates /is  regulated  by 

4.  Process 

has  object/ is  object  of 

has  output/ output  of 

has  result/results  from 

has  subprocess/is  subprocess  of 

has  process /is  process  in 

organizes /is  organized  by 

has  input/ is  input  to 

proposes/is  proposed  by 

depends  on/has  dependent 

concludes/is  concluded  by 

5.  Temporal 

has  step /is  step  in 
precedes  /  follows 

has  stage /is  stage  in 
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HPKMT  Authoring  System  Overview 

The  Web-based  HPKMT  Authoring  System  supports  the  online  creation  and 
maintenance  of  knowledge  mapping  tasks.  The  design  of  the  system  was  based  on 
the  work  of  Chung,  Baker  et  al.  (2002).  The  near-term  goal  of  the  authoring 
functionality  was  to  support  research  activities.  A  long-term  goal  was  to  develop  an 
interface  that  is  suitable  for  use  by  a  wide  audience  (e.g.,  trainers,  trainees, 
researchers,  course  managers). 


The  functionality  of  the  Authoring  System  is  outlined  in  the  home  page,  which 
displays  a  site  map  (see  Figure  1).  New  users  unfamiliar  with  the  system  are  advised 
to  click  on  Tasks  to  begin  creating  a  knowledge  mapping  task.  From  the  Tasks  page, 
a  user  can  add  all  of  the  elements  to  make  a  complete  knowledge  mapping  task. 


13  Knowledge  Mapper  -  Microsoft  Internet  Explorer 
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:  You  can  navigate  the  Knowledge  Mapping  Tool  Authoring  System  website  by  clicking  on  any  of  the  words  above.  If  you 
are  a  new  user,  we  suggest  that  you  follow  the  path  that  starts  on  TASKS.  > 


l  ®  Internet 


Figure  1.  Home  page  of  HPKMT  Authoring  System. 
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Users  can  click  on  any  of  the  words  in  the  graphic  to  perform  a  specific  action, 
such  as  modifying  a  concept  set  or  removing  an  icon  set.  Each  of  the  boldfaced 
words  is  a  major  category  (Tasks,  Courses,  Task  Components,  Users,  and  Reports); 
access  to  these  is  provided  at  the  top  of  every  page  in  the  Authoring  System. 

To  organize  the  set  of  tasks,  users  first  create  a  course.  A  course  consists  of  one 
or  more  knowledge  mapping  tasks.  Each  mapping  task  has  properties  associated 
with  it,  such  as  the  set  of  concepts  and  links,  the  set  of  icons  used,  and  the 
background  of  the  canvas.  Tasks  can  be  created  by  using  existing  concepts  and  links 
or  by  creating  new  ones  or  augmenting  existing  sets.  Users  enter  information  about 
trainees  and  create  login  IDs  for  them  to  access  the  HPKMT  software.  Once  trainees 
are  entered  in  the  system,  they  are  assigned  tasks.  Individual  trainees  can  be  put  into 
groups  to  make  administration  functions  (such  as  scoring  and  assigning  tasks) 
easier.  In  the  following  sections,  each  major  functional  area  is  discussed  with 
accompanying  screen  shots. 


Tasks 

To  create  a  new  task,  users  go  to  Tasks  >  Add.  In  this  page  (see  Figure  2),  users 
provide  a  task  name  and  select  the  course  to  which  this  task  belongs.  For  Task  Type, 
users  set  the  kind  of  knowledge  representation  of  the  task;  users  can  select  from 
declarative,  procedural,  and  physical.  Task  Mode  is  used  to  designate  whether  the 
student  can  only  select  concepts  and  relationships,  only  type  in  concepts  and 
relationships,  or  do  both. 
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H  Knowledge  Mapper ::  tasks  ::  add  -Microsoft  Internet  Explorer 
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Figure  2.  Add  Task  page. 


Users  can  either  select  existing  concept,  relationship,  and  icon  sets  and 
backgrounds  to  use  in  their  task,  modify  the  existing  entities,  or  create  their  own. 
Users  can  view  the  contents  of  existing  entities  by  clicking  on  List  (see  Figure  3). 
When  users  are  done  setting  the  specifications  of  their  task,  they  click  on  Save  and 
are  taken  to  the  Assign  page. 
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'H  Knowledge  Mapper ::  tasks  ::  add  -  Microsoft  Internet  Explorer 
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Figure  3.  List  Concept  set  view. 


Courses 

By  bundling  multiple  tasks  into  a  course,  users  can  organize  tasks  in  the 
Authoring  System.  A  task  is  assigned  to  a  course  from  the  Tasks  page;  a  course  can 
be  created  from  that  page  or  through  the  Courses  link.  In  addition,  courses  can  be 
modified,  deleted,  or  viewed  through  the  Courses  main  page. 

Task  Components 

Tasks  are  necessarily  made  up  of  three  components:  a  concept  set,  a 
relationship  set,  and  an  icon  set.  In  addition,  a  background  can  be  selected,  usually 
for  tasks  where  placement  of  nodes  is  as  important  or  more  important  than  the 
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relationships  between  them.  If  no  background  is  selected,  the  drawing  canvas  is 
white. 

Users  who  want  to  manipulate  individual  task  elements  instead  of  the  entire 
task  can  go  to  Task  Components  and  select  the  element  (Concept  Sets, 
Relationship  Sets,  Icon  Sets,  or  Backgrounds)  they  wish  to  work  on. 

Task  components  are  independent  of  tasks,  so  they  can  be  reused  across  tasks. 
Users  should  be  aware  however  that  changing  a  task  component  that  is  already  in 
use  in  a  task  could  have  adverse  effects. 

Concept  Sets 

To  create  a  new  concept  set,  users  click  on  Task  Components  >  Concept  Sets  > 
Add.  Users  will  see  a  form  as  presented  in  Figure  4.  Users  name  the  task  and  also 
provide  a  description,  which  can  be  particularly  useful  if  there  are  multiple  versions 
of  a  concept  set  for  one  subject  area  or  task  (e.g.,  the  user  can  note  the  date  modified, 
version  number,  or  notes  about  edited  concepts). 

Users  then  type  or  paste  the  concepts  into  the  text  box — concepts  are  separated 
by  carriage  returns. 

If  users  want  the  Authoring  System  to  alphabetize  the  concepts  that  were 
entered,  they  can  check  the  box  for  "Alphabetical  Order?"  Otherwise,  concepts  will 
be  presented  in  the  HPKMT  in  the  order  they  were  entered  by  the  user. 

Users  can  also  specify  if  they  wish  to  have  concepts  grouped  within  the  concept 
list.  This  means  in  the  HPKMT  when  the  concept  list  is  displayed,  concepts  can  be 
set  apart  from  others  by  a  thin  gray  line.  Users  may  wish  to  use  this  function  for 
tasks  dealing  with  different  categories  of  entities — for  example,  a  concept  list  might 
separate  people,  places,  and  events,  or  for  a  physiology  task,  concepts  related  to  the 
digestive  system  might  be  set  apart  from  ones  related  to  the  circulatory  system.  To 
turn  on  concept  grouping,  users  need  to  check  the  box  next  to  "Group  Category?" 
and  also  set  apart  the  concept  groups  by  entering  separators  in  the  concept  list — two 
or  more  consecutive  hyphens  on  their  own  line  constitute  a  separator. 

After  concepts  have  been  entered  and  preferences  for  the  concept  set  have  been 
checked,  the  user  can  click  Add  to  save  the  concept  set. 
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Figure  4.  Add  Concept  Set  page. 


Users  can  click  on  Concept  Sets  >  View  All  to  review  existing  concept  sets. 
From  that  page,  they  can  also  remove  and  modify  concept  sets,  or  they  can  also  click 
on  the  Remove  or  Modify  links  near  the  top  right-hand  corner  of  the  page.  (The 
options  to  add,  remove,  modify,  and  view  all  are  available  for  all  of  the  functions  in 
the  Authoring  System  except  for  Reports.) 


Relationship  Sets 

Relationship  sets  function  similarly  to  concept  sets.  In  the  Task  Components  > 
Relationship  Sets  >  Add  page,  users  enter  a  relationship  set  name,  description,  and 
the  set  of  relationships. 


In  addition,  if  users  want  to  create  a  task  that  does  not  require  students  to  label 
all  links  between  concepts,  users  can  check  the  box  next  to  "Allow  Label-less  links?" 
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If  this  box  is  checked,  in  the  HPKMT,  the  student  will  see  the  list  of  relationships  (if 
any  have  been  entered)  and  an  additional  option  that  says  "[no  label]."  If  the  student 
selects  "[no  label]"  for  a  relationship  in  the  map,  there  will  only  be  an  arrow 
connecting  the  two  concepts  and  no  accompanying  link  label. 
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Figure  5.  Add  Relationship  Set  page. 


Icon  Sets 

Icon  sets  are  used  to  specify  the  concept  symbols  served  up  in  the  HPKMT.  The 
most  commonly  used  icon  set  represents  a  concept  as  a  blue  rectangle  with  the 
concept  label  within  the  box.  If  preferred,  however,  users  can  upload  their  own  icons 
and  create  icon  sets  tailored  to  their  tasks. 

The  specifications  for  icon  files  are  on  the  Task  Components  >  Icon  Sets  >  Add 
page  (see  Figure  6).  Users  upload  icons  from  their  personal  computers  to  the  library 
by  clicking  on  the  Browse  button  and  locating  the  picture  file.  After  users  click  on 
Add,  the  picture  file  is  uploaded  to  the  library  and  appears  in  the  window  with  the 
scroll  bar  at  the  bottom  of  the  page. 
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One  or  more  icons  are  then  checked  off  to  be  added  to  the  new  icon  set;  after 
the  user  clicks  Add  at  the  bottom  of  the  page,  the  icon  set  is  available  to  be  used  in  a 
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The  icons  should  be  16x16  pixels,  and  no  more  than  5  Kb. 


Icon  set  name: 
Description: 

Add  Icons: 


□ 

Browse...  <  add  > 


(browse  from  your 
computer  or  choose 
from  our  library  to 

set  up  your  icon  set  .. 

list  on  the  right  hand  side)  ^nd  i  or  choose  from  the  library: 


1.  □  ^Concept 

2.  □  Ostep 

3.  □  ©>Car 

4.  □  J^Plane 

5.  □  JSShip 

6.  □  ® T rain 

7.  □  ^Grid 

o  I  I  P"V/i  ir  ri 


Figure  6.  Add  Icon  Set  page 
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Backgrounds 

The  default  for  the  canvas  background  in  the  HPKMT  is  a  white  mapping 
space,  but  some  tasks  may  require  users  to  place  concepts  or  nodes  on  a 
background.  Users  can  upload  picture  files  to  be  used  for  canvas  backgrounds  in 
Task  Components  >  Backgrounds>  Add  (see  Figure  7). 

Specifications  on  how  to  upload  backgrounds  are  provided  at  the  top  of  the 
page.  The  procedures  for  uploading  backgrounds  to  the  library  and  saving  them  are 
identical  to  those  for  icon  sets. 
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home  tasks  courses  > task  components  users  reports 

.  . .  [add]  [remove]  l  modify]  fvfe  wa  II  ] 

concept  sets  relationship  sets  icon  sets  backgrounds 

Instructions  on  how  to  add  backgrounds:  The  image  you  upload  is  not  resizable,  so  it  is 
going  to  show  up  on  the  page  in  the  same  size  you  upload  it.  We  recommend  that  you 
upload  a  . gif  or  .jpg  image,  with  72  dpi  resolution  (screen  resolution).  The  backgrounds 
should  be  (1)  640  x480  pixels  (for  screen  resolution  800  x  600),  (2)  800  x  600  (for  screen 
resolution  1024  x768),  or  (3)  1024  x  768  pixels  (for  screen  resolution  1152  x864). 

Background  name: 


Description: 


Add  new  background:  Browse...  I  <  add  > 


(browse  from  your  computer  or 
choose  from  our  library. ) 


and  /  or  choose  from  the  library: 

1.  D  O  Concept 

2.  D  Ostep 

3.  D  ^Car 

4.  □  J^Plane 

5.  □  i2Ship 

6.  □  IS  Train 

7.  □  ^Gnd 

o  r— i  rrv.-TT c*  a 


<  submit  > 


Internet 


Figure  7.  Add  Backgrounds  page. 
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Users 

In  order  to  use  the  HPKMT  and  create  knowledge  maps,  a  person  needs  to  be 
set  up  as  a  user  with  a  login  and  password.  The  form  to  add  users  (see  Figure  8) 
allows  for  the  easy  entry  of  multiple  users;  information  can  be  copied  from  a  text  file 
and  pasted  into  the  form.  Commas  separate  the  required  elements  of  the  user's 
profile  (first  name,  last  name,  user  name,  and  password).  The  user  name  is  a  unique 
login  to  access  the  HPKMT — e-mail  addresses  can  be  used  to  avoid  potential 
conflicts  with  existing  logins.  Each  user's  information  is  separated  by  a  carriage 
return. 


Figure  8.  Add  Users  page. 


In  Users  >  Modify  (see  Figure  9),  a  fuller  user  profile  is  provided.  If  a  user  is  to 
be  designated  an  expert  (someone  who  will  create  a  criterion  map  for  a  task).  User 
Type  needs  to  be  changed  from  Student  to  Expert.  In  this  page,  user  information  can 
be  edited  to  correct  misspellings  or  to  change  passwords  as  well. 
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3  Knowledge  Mapper  -  users ::  modify  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  Tools  Help 


|  Back  '■  ^  jji 


X*  jjSf  f  J---  Search  Favorites  Media  ^0 


y  l  '  iZ7i ' 


Address 


jv  Go  Links 


home  tasks  courses  task  components  >  users  reports 


.  [add]  [removejfmodffy]  |vte  wall] 


users  groups 


Select  User:  Example  ,  Student 


User:  Example  ,  Student  ( student(§>example  [ 


First  name:  Student 


Last  name:  Example 


Username:  student@example 


Password:  •••• 


User  type:  Student 


Language:  American  English  y 


Time  zone:  Pacific  Standard  Time  Iv 


<  modify  > 


Figure  9.  Modify  User  page. 


Internet 


Groups 

Groups  can  be  created  to  facilitate  user  management.  Individual  users  must  be 
selected  when  assigning  tasks  or  generating  score  reports;  grouping  users  can  make 
these  functions  less  cumbersome  by  eliminating  the  need  to  click  on  multiple  users 
or  to  perform  the  same  task  repeatedly. 
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Reports 


Several  different  scoring  schemes  have  been  developed,  and  continue  to  be 
developed,  for  knowledge  maps.  The  HPKMT  Authoring  System  is  flexible  enough 
to  allow  for  the  addition  of  new  schemes  as  they  are  conceived.  Currently  there  are 
three  scoring  schemes  (two  for  declarative  and  procedural  maps,  one  tailored  for  a 
specific  physical  mapping  task)  available  in  the  Reports  section  (see  Figure  10),  but 
new  ones  can  be  integrated  seamlessly. 


H  Knowledge  Mapper ::  reports  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  Tools  Help 


Address 


*3  ©  Search  ^Favorites  Media  <0  0* 

|M:i  |3  Go  !  Links 


home  tasks  courses  task  components  users  >  reports 


reports  maps 


1.  Semantic  Scoring  Report 

2.  Structural  Scoring  Report 

3.  Target  Error  Scoring  Report 


Internet 


Figure  10.  Reports  page. 


Semantic  Scoring 

Semantic  scoring  compares  the  propositions  in  a  student's  map  to  those  in  an 
expert's  map.  In  the  Authoring  System,  users  select  from  a  pull-down  menu  both  the 
task  and  the  expert  who  created  the  criterion  map  (see  Figure  11).  Then,  a  score 
report  can  be  generated  for  all  users  who  created  a  map  for  that  task  by  checking  the 
box  next  to  "All  Users,"  or  if  a  score  report  is  needed  for  only  a  subset  of  students, 
individual  users  or  groups  of  users  can  be  selected  by  using  control-click. 
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H  Knowledge  Mapper ::  reports  -  Microsoft  Internet  Explorer 


tasks  courses  task  components  users  >  reports 


[remove 


reports  maps 


Semantic  Scoring  Report 


All  Users  :  [P] 


Figure  11.  Semantic  scoring  specification  page. 


At  the  bottom  of  the  semantic  scoring  report  specification  page  are  five 
checkboxes  (see  Figure  12).  These  are  used  to  set  the  types  of  scores  to  be  retrieved. 
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» 


Descriptions  of  each  type  (Exact,  Directionless,  Linkless,  Linkless  with  Direction, 
and  Synonyms)  are  below. 

Exact 

Direction  Less 
Linkless 

Linkless  With  Direction 
Synonyms 


<  submit  > 


§g 

|®  Done 

.  '  . 

$  Internet  | 

Figure  12.  Semantic  scoring  dimensions. 


Scoring 
Schemes  : 


F 

FI 


Exact  scoring  produces  the  number  of  propositions  in  the  student  map  that 
exactly  match  propositions  in  that  expert's  map. 

Directionless  scoring  generates  the  number  of  propositions  in  the  student  map 
that  match  propositions  in  the  expert  map  if  you  disregard  the  arrow  direction  but 
take  into  account  the  relationship  label. 

Linkless  scoring  produces  the  number  of  propositions  in  the  student  map  that 
match  propositions  in  the  expert  map  if  you  disregard  both  the  arrow  direction  and 
the  relationship  label. 

Linkless  with  Direction  scoring  produces  the  number  of  propositions  in  the 
student  map  that  match  propositions  in  the  expert  map  if  you  disregard  the 
relationship  label  but  take  into  account  the  arrow  direction. 

Synonym  scoring  produces  the  number  of  propositions  in  the  student  map  that 
match  propositions  in  the  expert  map  when  the  links  in  both  the  student  and  expert 
maps  are  compared  on  a  set  of  links  (i.e.,  the  synonym  set).  Synonym  scoring  is  a 
more  lenient  method  that  yields  a  match  if  there  exists  an  intersection  between  the 
set  of  synonyms  in  the  student  proposition  and  the  set  of  synonyms  in  the  expert's 
proposition. 

For  example,  suppose  the  links  causes,  leads  to,  and  influences  were  defined  to  be 
synonyms.  A  comparison  between  a  student  proposition  and  an  expert  proposition 
would  yield  a  match  if  both  the  expert  and  student  propositions  had  any  of  the 
synonyms  (i.e.,  causes,  leads  to,  or  influences). 
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Figure  13.  Example  expert  and  student  propositions. 


To  illustrate  the  different  types  of  scores.  Figure  13  shows  examples  of 
propositions  from  an  expert  and  from  a  student. 

Under  Exact  scoring,  this  student  would  only  get  1  point,  for  the  proposition 
heart  >  pumps  >  blood.  The  is  the  only  proposition  that  has  an  exact  match  in  the 
expert  map,  meaning  both  the  arrow  direction  between  the  two  nodes  and  the 
relationship  label  are  the  same. 

Under  Directionless  scoring,  the  student  would  get  2  points.  The  proposition 
heart  >  pumps  >  blood  gets  1  point,  and  another  point  is  given  for  circulatory  system 
>  made  up  of  >  body  because  the  nodes  are  connected  by  the  same  link  the  expert 
used,  except  the  arrow  direction  is  reversed  in  the  expert  map. 

Under  Linkless  scoring,  the  student  would  get  4  points  because  arrow  direction 
and  link  label  are  both  disregarded.  The  student  gets  a  point  for  relating  two  nodes 
in  any  way  that  the  expert  has  connected.  The  propositions  heart  >  pumps  >  blood 
and  circulatory  system  >  made  up  of  >  body  both  get  a  point  each.  In  addition,  the 

propositions  blood  >  part  of  >  body  and  heart  >  maintains  >  circulatory  system  also 
get  a  point  each — this  scoring  scheme  has  found  a  match  with  the  expert 
propositions  that  linked  blood  with  body  and  heart  with  circulatory  system. 

Under  Linkless  with  Direction  scoring,  the  student  would  get  3  points.  The 
proposition  heart  >  pumps  >  blood  gets  1  point,  and  points  are  given  for  blood  >  part 
of  >  body  and  heart  >  maintains  >  circulatory  system  because  the  relationship  labels 
are  disregarded,  but  the  arrow  direction  of  the  relationships  is  taken  into  account. 
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Under  Synonym  scoring,  suppose  the  links  controls  and  maintains  were  defined 
as  synonyms.  There  would  be  a  match  between  the  student  and  expert  on  the 
proposition  heart  >  maintains  >  circulatory  system. 

Structural  Scoring 

Structural  scoring  is  the  second  type  of  scoring  scheme  supported  by  the 
Authoring  System.  We  are  currently  developing  measures  that  report  on  the 
configural  properties  of  the  maps.  For  example,  for  each  concept,  the  number  of  fan- 
in  and  fan-out  connections  are  computed  as  well  as  the  corresponding  list  of  parent 
and  descendant  concepts,  respectively.  For  each  concept,  reachability  is  determined 
(i.e.,  the  set  of  concepts  that  could  be  reached  from  the  current  concept). 

The  fan-in  value  helps  identify  concepts  that  are  sinks  and  the  fan-out  value 
helps  identify  concepts  that  are  sources.  Reachability  is  a  measure  of  the  network 
connectivity — the  higher  the  number  of  concepts  that  are  reachable  from  a  node,  the 
more  interconnected  the  network. 

To  obtain  structural  map  data,  users  select  the  option  for  structural  scoring 
from  the  Reports  page,  and  then  choose  the  knowledge  mapping  task  for  which  they 
want  scores.  They  are  then  directed  to  a  page  that  lists  all  the  maps  produced  for 
that  task.  Users  can  either  select  multiple  maps  and  download  one  Excel  file  with 
structural  data  for  all  the  selected  maps,  or  click  on  the  S.  Report  link  next  to  the 
map  name  to  download  structural  map  data  for  only  that  map  (see  Figure  14). 
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File  Edit  View  Favorites  Tools  Help 


("i  /  '  Search  Cv  Favorites 


•Media  0’  Kjjs-  !.*  j 

ivj^JjGo  links 


tasks  courses  task  components  users  >  reports 

___ _ _ _ _[rernove][vtew  sf!J 

reports  maps 

Structural  Score  Report 


Task :  Physiology 


Demo  Physiology 

2,  [p~  DemoPhysiology8804 

3,  [pT  test 

Demo 


<  submit  > 


Clear  All  Mark  All 


[S  Report] 


[S.  Report] 


[S.  Report] 


[$.  Report] 


$  Internet 


Figure  14.  Structural  Score  Report  page. 


Task-Specific  Scoring  Schemes 

The  third  scoring  scheme  currently  available  through  the  Authoring  System  is 
specific  to  the  Target  Error  task.  This  scheme  produces  scores  based  on  the  physical 
placement  of  nodes  on  the  canvas  and  employs  algorithms  tailored  to  the  task  to 
produce  scores  (see  Chung,  Michiuye  et  alv  2002,  for  an  in-depth  description  of  this 
scoring  method). 
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Discussion  and  Next  Steps 

Our  design  goal  for  the  authoring  system  was  to  develop  a  simple,  user- 
friendly  interface  that  would  provide  essential  functionality  with  a  minimal  number 
of  clicks.  The  authoring  system  is  intended  to  allow  a  diverse  set  of  users  to  create, 
modify,  adapt,  and  reuse  knowledge  mapping  tasks.  Various  scoring  options 
provide  information  on  student  performance  on  different  dimensions  and  different 
levels  of  stringency.  The  system  stores  student  data  and  thus  performance  can  be 
monitored  over  time. 

We  are  currently  conducting  a  usability  study  of  the  authoring  system  with  a 
range  of  user  types  who  we  expect  to  be  using  this  system  (i.e.,  naive  user,  task- 
familiar  but  unfamiliar  with  authoring).  A  continuing  activity  is  to  refine  and  extend 
the  scoring  to  incorporate  new  developments  as  they  mature.  For  example,  a  highly 
desirable  feature  is  to  incorporate  scoring  of  student-typed  links.  We  have  made 
some  progress  toward  this  capability  but  lack  sufficient  data  to  fully  test  the 
algorithm.  We  also  plan  to  expand  the  reporting  capability  to  provide  simple  but 
useful  reports  for  potential  end-users  (e.g.,  summary  reports  for  groups  of  students, 
by  task,  by  individual). 
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